ALIGNING LARGE LANGUAGE MODELS

Comparing Bad Apples to Good Oranges Aligning Large Language Models via Joint Preference Optimization (2025.findings-acl)

Challenge:	Current methods to improve data quality are labor-intensive or prone to factual errors caused by LLM hallucinations.
Approach:	They propose a method which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence.
Outcome:	The proposed approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques.

Copied to clipboard

Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, Aditya Grover

Causal Direct Preference Optimization for Language Model Alignment (2026.findings-eacl)

Challenge:	Recent studies have shown that acquiring human preferences by comparing generations is not effective for large language models.
Approach:	They propose a preference optimization objective that elicits preferences jointly over the instruction-response pairs.
Outcome:	The proposed approach outperforms prior preference optimizations by 5.2% and 3.3% in summarization and open-ended dialogue datasets.

Copied to clipboard

Uyen Le, Thin Nguyen, Toan Nguyen, Toan Doan, Trung Le, Bac Le

Fine-Tuning Language Models with Reward Learning on Policy (2024.naacl-long)

Challenge:	Empirical evaluations show that CDPO surpasses DPO-based baselines by achieving unbiased fine-tuning through causal reasoning.
Approach:	They propose a framework that incorporates causal inference principles to mitigate the influence of confounders and sharpen the signal of genuine human preferences.
Outcome:	The proposed framework preserves the tractability of direct optimization while enhancing robustness to spurious correlations and annotation biases.

Copied to clipboard

Hao Lang, Fei Huang, Yongbin Li

RELIC: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples (2025.findings-emnlp)

Challenge:	Reinforcement learning from human feedback (RLHF) is an effective approach to align large language models (LLMs) to human preferences.
Approach:	They propose a framework that refines a reward model using policy samples to keep it on-distribution.
Outcome:	The proposed framework outperforms the state-of-the-art on three benchmark datasets showing that it can learn robust representations of policy samples.

Copied to clipboard

Soumya Suvra Ghosal, Vaibhav Singh, Akash Ghosh, Soumyabrata Pal, Subhadip Baidya, Sriparna Saha, Dinesh Manocha

Low-Resource NLP

Challenge:	a new reward model for low-resource Indic languages is proposed . a preference-based training approach is prohibitively expensive, authors say .
Approach:	a new in-context learning framework is proposed to train a retriever to select in-constext examples from low-resource Indic languages.
Outcome:	a new in-context learning framework for reward modeling in low-resource Indic languages is developed . the proposed framework outperforms existing examples on three preference datasets .

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards (2026.findings-acl)

Copied to clipboard

Yupeng Chang, Yuan Wu, Yi Chang

Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty (2026.findings-acl)

Challenge:	Critic-free reinforcement learning with verifiable rewards (RLVR) is a practical paradigm for aligning Large Language Models.
Approach:	They propose a framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments.
Outcome:	Experiments show that RLVR improves training stability and performance compared to critic-based methods . compared with other approaches, RL VR improves in cold-start regimes with binary verifiers .

Copied to clipboard

Yao Xiao, Jung-jae Kim, Roy Ka-Wei Lee, Lidong Bing

Beyond Excess and Deficiency: Adaptive Length Bias Mitigation in Reward Models for RLHF (2025.findings-naacl)

Challenge:	incorporating difficult prompts into training fails to enhance overall performance, e.g., as prompt difficulty decreases.
Approach:	They investigate how prompts of varying difficulty influence self-play preference optimization . they use the reward of sampled responses of a prompt as a proxy for its difficulty .
Outcome:	The proposed model improves on difficult prompts and easy prompts, but fails to train on difficult ones and learns from failures.

Copied to clipboard

Yuyan Bu, Liangyu Huo, Yi Jing, Qing Yang

Challenge:	Existing efforts to mitigate length bias in reward models have decreased accuracy . achieving an automatic proxy that perfectly replicates human judgment is challenging in practice.
Approach:	They propose an adaptive approach that dynamically adjusts the influence of response length in reward evaluations according to the context of the query.
Outcome:	The proposed approach reduces unnecessary verbosity while improving overall response quality.

CodecLM: Aligning Language Models with Tailored Synthetic Data (2024.findings-naacl)

Copied to clipboard

Zifeng Wang, Chun-Liang Li, Vincent Perot, Long Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister

RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution (2025.emnlp-main)

Challenge:	Recent work on generating diverse instructions and applying LLM to increase instruction complexity neglects downstream use cases.
Approach:	They propose a framework for generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs.
Outcome:	Experiments on four open-domain instruction using the proposed framework validate the effectiveness of CodecLM over the current state-of-the-art.

Copied to clipboard

Jiahui Li, Lin Li, Tai-Wei Chang, Kun Kuang, Long Chen, Jun Zhou, Cheng Yang

CYCLE-INSTRUCT: Fully Seed-Free Instruction Tuning via Dual Self-Training and Cycle Consistency (2025.emnlp-main)

Challenge:	Experimental results demonstrate the superiority of our approach to aligning large language models with human preferences.
Approach:	They propose a method that evaluates and assigns specific credit to each token using an off-the-shelf reward model.
Outcome:	The proposed method evaluates and assigns specific credit to each token using an off-the-shelf reward model.

Copied to clipboard

Zhanming Shen, Hao Chen, Yulei Tang, Shaolin Zhu, Wentao Ye, Xiaomeng Hu, Haobo Wang, Gang Chen, Junbo Zhao

Machine Translation

Challenge:	Existing methods for instruction tuning rely on expensive human-annotated seed data or powerful external teacher models.
Approach:	They propose a framework that achieves fully seed-free instruction tuning by employing a dual self-training loop where two models are bootstrapped solely from raw, unlabeled text.
Outcome:	The proposed framework outperforms seed-driven back-translation baselines and achieves comparable performance to strongly supervised methods.

ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning (2025.findings-acl)

Copied to clipboard

Yeyuan Wang, Dehong Gao, Rujiao Long, Lei Yi, Linbo Jin, Libin Yang, Xiaoyan Cai

Multimodality

Challenge:	Recent advances have extended DPO to multimodal scenarios, achieving strong performance.
Approach:	They propose to use a sentence-level preference optimization technique to optimize individual sentences for more precise preference optimization without additional models or parameters.
Outcome:	Experiments show that Adaptive Sentence-level Preference Optimization significantly improves the alignment of multimodal models.

CLHA: A Simple Yet Effective Contrastive Learning Framework for Human Alignment (2024.lrec-main)

Copied to clipboard

Feiteng Fang, Liang Zhu, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang

Aligning Large Language Models with Implicit Preferences from User-Generated Content (2025.acl-long)

Challenge:	Large language models (LLMs) have attracted considerable attention from academic and industrial communities due to their outstanding performance in various natural language processing tasks.
Approach:	They propose a Contrastive Learning Framework for Human Alignment to evaluate the noise within the data and dynamically adjust the training process.
Outcome:	The proposed framework surpasses other algorithms in terms of reward model scores, automatic evaluations, and human assessments on the widely used dataset "Helpful and Harmless"

Copied to clipboard

Zhaoxuan Tan, Zheng Li, Tianyi Liu, Haodong Wang, Hyokun Yun, Ming Zeng, Pei Chen, Zhihan Zhang, Yifan Gao, Ruijie Wang, Priyanka Nigam, Bing Yin, Meng Jiang

Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting (2025.findings-naacl)

Challenge:	Existing preference learning methods rely heavily on curated data from humans or advanced LLMs, which is costly and difficult to scale.
Approach:	They propose a framework that leverages implicit preferences in unlabeled user-generated content to generate preference data.
Outcome:	The proposed framework transforms user-generated content into user queries and generates responses from the policy model.

Copied to clipboard

Mohamed Salim Aissi, Clément Romac, Thomas Carta, Sylvain Lamprier, Pierre-Yves Oudeyer, Olivier Sigaud, Laure Soulier, Nicolas Thome

ToolRM: Towards Agentic Tool-Use Reward Modeling (2026.findings-acl)

Challenge:	Reinforcement learning (RL) is a promising approach for aligning large language models knowledge with sequential decision-making tasks.
Approach:	They propose to use a contrastive loss framework to analyze the sensitivity of LLMs to prompt formulations following RL training in a textual environment.
Outcome:	The proposed framework improves the model's robustness and generalization capabilities by minimizing the model’s internal representations and salient tokens.

Copied to clipboard

Renhao Li, Jianhong Tu, Yang Su, Yantao Liu, Fei Huang, Hamid Alinejad-Rokny, Derek F. Wong, Junyang Lin, Min Yang

Challenge:	lack of reliable reward models for tool-use tasks has limited progress toward agentic AI . recent advances in agentic artificial intelligence are driven by tool-using capabilities of large language models.
Approach:	They propose a pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling to build lightweight reward models.
Outcome:	The proposed model outperforms existing models on tool calling tasks with higher accuracy.

AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling (2026.acl-long)

Copied to clipboard

Yongliang Miao, Yangyang Liang, Mengnan Du

Challenge:	Existing reward models rely on a static pooling strategy to condense sequences into scalar scores, which is ill-suited for fine-grained discrimination.
Approach:	They propose a framework that jointly adapts representation and aggregation to address these limitations by integrating a static inductive bias with a representational mismatch.
Outcome:	Experiments on RM-Bench and JudgeBench show that AdaJudge outperforms strong off-the-shelf reward models and traditional pooling baselines.

Forward Knows Efficient Backward Path: Saliency-Guided Memory-Efficient Fine-tuning of Large Language Models (2025.acl-long)

Copied to clipboard

Yeachan Kim, SangKeun Lee

Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data (2024.findings-emnlp)

Challenge:	a number of fine-tuning approaches are available to improve performance of large language models.
Approach:	They propose a memory-efficient method to minimize memory associated with cached intermediate activations.
Outcome:	The proposed method minimizes memory associated with cached intermediate activations while preserving accuracy.

Copied to clipboard

Han Xia, Songyang Gao, Qiming Ge, Zhiheng Xi, Qi Zhang, Xuanjing Huang

TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models (2024.findings-emnlp)

Challenge:	Reinforcement Learning from Human Feedback (RLHF) relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and pose challenges in sample efficiency and stability.
Approach:	They propose an innovative framework that leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model’s responses.
Outcome:	The proposed framework matches and exceeds the effectiveness of Proximal Policy Optimization (PPO) in terms of convergence speed and alignment of model responses with human preferences.

Copied to clipboard

Chen Zhang, Chengguang Tang, Dading Chong, Ke Shi, Guohua Tang, Feng Jiang, Haizhou Li

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models (2025.acl-long)

Challenge:	Mainstream approaches to aligning large language models heavily rely on human preference data.
Approach:	They propose a framework that fine-tunes a policy model using pairwise feedback data automatically mined from its outputs.
Outcome:	The proposed framework outperforms the base model with an average win rate of 69.7% across seven conversational or instruction-following datasets.

Copied to clipboard

Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, Qing He

Self-Generated Critiques Boost Reward Modeling for Language Models (2025.naacl-long)

Challenge:	Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences.
Approach:	They propose a novel algorithm that uses multiple-gradient descent to optimize LLMs with diverse preferences to maximize trade-offs between objectives.
Outcome:	The proposed approach incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user’s specific needs.

Copied to clipboard

Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Rui Hou

KNN-Instruct: Automatic Instruction Construction with K Nearest Neighbor Deduction (2024.emnlp-main)

Challenge:	Existing reward models produce scalar scores and struggle to incorporate critiques in a natural language format.
Approach:	They propose a framework that predicts critiques and rewards using self-generated critiques without extra supervision.
Outcome:	The proposed framework improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges.

Copied to clipboard

Jianshang Kou, Benfeng Xu, Chiwei Zhu, Zhendong Mao

Direct Preference Optimization with an Offset (2024.findings-acl)

Challenge:	Existing methods for generating synthetic instructions for large language models suffer from stale distribution and scalability.
Approach:	They propose a method which incorporates KNN deduction to produce meaningful new instructions by summarizing and learning from existing ones.
Outcome:	The proposed method outperforms all 7B models on the LMSYS leaderboard.

Copied to clipboard

Afra Amini, Tim Vieira, Ryan Cotterell

Learning Temporally-Aware Sample Weights for Preference Optimization (2026.findings-acl)

Challenge:	Direct preference optimization (DPO) fine-tunes language models with human preferences . but not all preference pairs are equal; sometimes, the preferred response is only slightly better than the dispreferred one.
Approach:	They propose a generalization of direct preference optimization that does not treat every preference pair equally during fine-tuning.
Outcome:	The proposed algorithm outperforms DPO on tasks with limited preference pairs . it requires the difference between likelihood of preferred and dispreferred response to be greater than offset value .

Copied to clipboard

Mengyang Li, Xudong Zhou, Pinlong Zhao

Challenge:	Existing methods for preference optimization rely on static functions of instantaneous model states and ignore temporal learning dynamics.
Approach:	They propose a framework that meta-learns adaptive weights using three temporal features: reward margin evolution, learning volatility, and reference deviation.
Outcome:	The proposed framework achieves statistically significant improvements over baselines on models ranging from 7B to 70B parameters.

Aligning Large Language Models by On-Policy Self-Judgment (2024.acl-long)

Copied to clipboard

Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, Youngjae Yu

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)

Challenge:	Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model for on-policy learning.
Approach:	They propose a new alignment framework that does on-policy learning and is parameter efficient . they propose Judge-augmented Supervised Fine-Tuning to train a single model to act as a policy and a judge.
Outcome:	The proposed framework outperforms baselines in preference benchmarks and rejecting sampling by itself improves performance without additional evaluator.

Copied to clipboard

Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang

Explainability & Interpretability in NLP

Challenge:	Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences.
Approach:	They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data.
Outcome:	The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark .

BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment (2024.emnlp-main)

Copied to clipboard

Wenda Xu, Jiachen Li, William Yang Wang, Lei Li

LimaCost: Data Valuation for Instruction Tuning of Large Language Models (2025.findings-emnlp)

Challenge:	Existing offline DAP methods for aligning large language models with human preference are computationally expensive due to their two-stage training pipeline that consists of a reward modeling phase.
Approach:	They propose to align large language models to human desiderata from offline preference datasets by using an online approach.
Outcome:	The proposed approach improves performance across a wide range of tasks when training with the same amount of preference data.

Copied to clipboard

Hyeonseok Moon, Jaehyung Seo, Seonmin Koo, Jinsung Kim, Young-kyoung Ham, Jiwon Moon, Heuiseok Lim

Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms (2026.findings-acl)

Challenge:	Instruction tuning is an effective approach for aligning large language models with human intentions.
Approach:	They propose a data quality measure that exhibits a strong correlation with model performance.
Outcome:	The proposed measure exhibits a strong correlation with model performance.

Copied to clipboard

Zeguan Xiao, Yun Chen, Jian Yang, Guanhua Chen, Ke Tang

Challenge:	Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, including instruction following, mathematical problem solving, and coding generation.
Approach:	They propose a method that truncates both preferred and dispreferred responses to match the shorter one’s length.
Outcome:	The proposed approach improves over standard implementations and achieves 11.8 points in AlpacaEval 2 and overall improvements across downstream tasks.

Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch (2025.acl-long)

Copied to clipboard

Xueru Wen, Jie Lou, Zichao Li, Yaojie Lu, XingYu XingYu, Yuqiu Ji, Guohai Xu, Hongyu Lin, Ben He, Xianpei Han, Le Sun, Debing Zhang

Challenge:	Existing Chinese resources are small in scale and limited to specific domains, making them insufficient for LLM post-training.
Approach:	They propose a Chinese-annotated reward model and a preference dataset to address this gap . they evaluate Chinese RMs on CheemsBench and construct an RM that captures human preferences .
Outcome:	The proposed RM achieves state-of-the-art performance on CheemsBench and CheeMePreference.

Better Alignment with Instruction Back-and-Forth Translation (2024.findings-emnlp)

Copied to clipboard

Thao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt, Jason Weston, Luke Zettlemoyer, Xian Li

What Do LLMs Learn First? Asymmetric Learning Dynamics of Input Complexity and Output Ambiguity in Preference Alignment (2026.acl-long)

Challenge:	et al., 2023) proposes a method to improve instruction-tuning data . e.g., we generate synthetic instructions using the backtranslation approach .
Approach:	They propose a method to improve instruction-tuning data using web-based inputs . they generate synthetic instructions using the backtranslation approach and filter the generated data .
Outcome:	The proposed method improves the quality of instruction-tuning data based on preprocessed texts . it yields better AlpacaEval win rates than direct distillation .

Copied to clipboard

Mengyang Li, Jingwen Wang, Pinlong Zhao

Challenge:	Existing methods treat all preference pairs uniformly during training.
Approach:	They propose a training framework that maintains separate, adaptive pacing schedules for each dimension.
Outcome:	The proposed training framework outperforms curriculum baselines by 2.1% and 0.21 points . it achieves 42.3% length-controlled win rate on AlpacaEval 2.0 and 7.66 on MT-Bench .

DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging (2024.emnlp-main)

Copied to clipboard

Tzu-Han Lin, Chen-An Li, Hung-yi Lee, Yun-Nung Chen

From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment (2025.acl-long)

Challenge:	Modern large language models (LLMs) showcase impressive capabilities across various tasks with aligning their behavior with human preferences.
Approach:	They propose a framework that integrates domain-specific knowledge into a general reward model by model merging.
Outcome:	The proposed framework improves performance across different benchmarks and provides detailed analysis showing the effects of model merging.

Copied to clipboard

Bin Xie, Bingbing Xu, Yige Yuan, Shengmao Zhu, Huawei Shen

Summarization

Challenge:	Existing approaches to align large language models with human preferences suffer from inconsistent scoring and suboptimal alignment.
Approach:	They propose a dual-consistency framework that aligns partial sequences with human preferences.
Outcome:	The proposed framework significantly reduces granularity discrepancies and improves GPT-4 evaluation scores.

IPO: Your Language Model is Secretly a Preference Classifier (2025.acl-long)

Copied to clipboard

Shivank Garg, Ayush Singh, Shweta Singh, Paras Chopra

AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs (2025.acl-long)

Challenge:	Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences, but it often incurs significant computational and financial costs due to its reliance on training external reward models or human-labeled preferences.
Approach:	They propose an alternative approach that leverages generative LLMs as preference classifiers to reduce the dependence on external reward models or human-labeled preferences.
Outcome:	The proposed approach reduces the dependence on external reward models or human-labeled preferences by using generative LLMs as preference classifiers.

Copied to clipboard

Nicholas E. Corrado, Julian Katz-Samuels, Adithya M Devraj, Hyokun Yun, Chao Zhang, Yi Xu, Yi Pan, Bing Yin, Trishul Chilimbi

DynamicFocalPO: Adaptive Focusing Strategy for Preference Optimization (2026.findings-acl)

Challenge:	Existing approaches to align large language models rely on large ablation studies, heuristics, or human intuition to produce models with strong performance across tasks.
Approach:	They propose an algorithm that mixes datasets during LLM training to balance performance across multiple tasks.
Outcome:	The proposed algorithm outperforms existing methods on multitask alignment setups and achieves convergence rate of O(1/T) in the convex case.

Copied to clipboard

Shu Zhou, Junan Chen, Rui Ling, Xin Wang, Tao Fan, Hao Wang

Debiasing Reward Models via Causally Motivated Inference-Time Intervention (2026.acl-long)

Challenge:	Recent preference optimization algorithms such as Direct Preference Optimization (DPO) have become prevalent for aligning large language models with human preferences.
Approach:	They propose a preference optimization algorithm that introduces a modulating factor that down-weighs misranked preference pairs and employs focusing strategy that adapts over the course of training.
Outcome:	Experiments show that DynamicFocalPO surpasses both DPO and FocalPO on benchmarks including Alpaca Eval 2.0 and Arena-Hard using Mistral-Base-7B and Llama-3-Instruct-8B.

Copied to clipboard

Kazutoshi Shinoda, Kosuke Nishida, Kyosuke Nishida

Robustness in NLP

Challenge:	Existing approaches for mitigating spurious features in RMs focus on response length . Existing methods focus on RM activation, resulting in performance trade-offs .
Approach:	They propose a method that uses neurons to suppress spurious features in RMs at inference time.
Outcome:	The proposed method reduces sensitivity to spurious features without inducing performance trade-offs on RM benchmarks.

Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity (2025.acl-long)

Copied to clipboard

Yupu Hao, Pengfei Cao, Zhuoran Jin, Huanxuan Liao, Yubo Chen, Kang Liu, Jun Zhao

Challenge:	Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools.
Approach:	They propose a key-point-based LLM evaluation method that mitigates biases by manually annotating key points for each test case and providing them to LLM as the reference.
Outcome:	The proposed method mitigates biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference.

Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality (2025.emnlp-main)

Copied to clipboard

Yuto Harada, Yusuke Yamauchi, Yusuke Oda, Yohei Oseki, Yusuke Miyao, Yu Takagi

Curiosity-Driven Reinforcement Learning from Human Feedback (2025.acl-long)

Challenge:	Recent advances in large language models (LLMs) have greatly improved natural language understanding and generation.
Approach:	They train a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks.
Outcome:	The results show that training–task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies.

Copied to clipboard

Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang

Challenge:	Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models with human preferences, but often at the cost of reduced output diversity.
Approach:	They propose a framework that incorporates intrinsic rewards for novel states alongside traditional sparse extrinsic rewards to optimize both output diversity and alignment quality.
Outcome:	The proposed framework achieves significant gains in diversity on multiple diversity-oriented metrics while maintaining alignment with human preferences comparable to standard RLHF.

Uncertainty-Aware Iterative Preference Optimization for Enhanced LLM Reasoning (2025.acl-long)

Copied to clipboard

Lei Li, Hehuan Liu, Yaxin Zhou, ZhaoYang Gui, Xudong Weng, Yi Yuan, Zheng Wei, Zang Li

Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)

Challenge:	Existing methods for enhancing the performance of large language models require expensive manual annotations.
Approach:	They propose an offline direct preference optimization method that collects preference pairs through iterative sampling and execution feedback to improve model confidence.
Outcome:	The proposed method improves performance on three reasoning tasks and shows a 3.6% improvement over the standard method.

Copied to clipboard

Zhang Zhang, Guhao Feng, Jian Guan, Di He, Wei Wu

Responsible & Trustworthy NLP

Challenge:	Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation.
Approach:	They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model.
Outcome:	The proposed approach matches or exceeds the performance of a fully online DPO.

Compatibility-Aware Dynamic Fine-Tuning for Large Language Models (2026.acl-long)

Copied to clipboard

Yucheng Zhou, Junwei Sheng, Qianning Wang, Jianbing Shen

𝒜3: Automatic Alignment Framework for Attributed Text Generation (2025.acl-long)

Challenge:	Recent work attributes optimization instability to the low probability of demonstrations being incompatible with the sample level.
Approach:	They propose a Dynamic Fine-Tuning extension of DFT that controls sample-level optimization variance.
Outcome:	The proposed model can generalize token-level stabilization to the sample level while remaining fully supervised and free of reward modeling.

Copied to clipboard

Yue Wang, Haoke Zhang, Juntao Li, Jinxiong Chang, Min Zhang

Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models (2026.acl-long)

Challenge:	Existing approaches of aligning large language models to follow user instructions can lead to undue emphasis on irrelevant documents, which in turn reduces the quality of responses.
Approach:	They propose to use a framework to automatically generate high-quality attributed query-response pairs for both supervised fine-tuning and preference optimization stages without human annotation.
Outcome:	The proposed framework can generate high-quality attributed query-response pairs without human annotation without human intervention.

Copied to clipboard

Tingchen Fu, Yafu Li, Jiawei Gu, Xiaoye Qu, Yu Cheng

Alignment Data Map for Efficient Preference Data Selection and Diagnosis (2026.findings-acl)

Challenge:	Recent advances in reasoning-oriented models have demonstrated impressive capabilities in mathematical reasoning, but their ability to adhere to user directives remains underexplored.
Approach:	They propose a benchmark to evaluate instruction-following in mathematical reasoning tasks.
Outcome:	The proposed model degrades in instruction adherence when generation length increases, but can partially recover obedience, despite increasing generation length.

Copied to clipboard

Seohyeong Lee, Eunwon Kim, Hwaran Lee, Buru Chang

Green & Sustainable NLP

Challenge:	constructing high-quality preference datasets faces scalability challenges due to prohibitive cost and complexity of human annotation.
Approach:	They propose a tool to identify and select effective preference data by LLM-as-a-judge, explicit reward model, and reference-based approaches.
Outcome:	The proposed tool reduces annotation costs while preserving alignment effectiveness.

Influence-based Online Experience Selection for Effective RLHF (2026.acl-long)

Copied to clipboard

Yifan Gong, Jing Yao, Xiting Wang, Xunlong Wang, Xiaoyuan Yi, Xing Xie